Navigation domain representation for interactive multiview imaging

Enabling users to interactively navigate through different viewpoints of a static scene is a new interesting functionality in 3D streaming systems. While it opens exciting perspectives towards rich multimedia applications, it requires the design of novel representations and coding techniques in order to solve the new challenges imposed by interactive navigation. Interactivity clearly brings new design constraints: the encoder is unaware of the exact decoding process, while the decoder has to reconstruct information from incomplete subsets of data since the server can generally not transmit images for all possible viewpoints due to resource constrains. In this paper, we propose a novel multiview data representation that permits to satisfy bandwidth and storage constraints in an interactive multiview streaming system. In particular, we partition the multiview navigation domain into segments, each of which is described by a reference image and some auxiliary information. The auxiliary information enables the client to recreate any viewpoint in the navigation segment via view synthesis. The decoder is then able to navigate freely in the segment without further data request to the server; it requests additional data only when it moves to a different segment. We discuss the benefits of this novel representation in interactive navigation systems and further propose a method to optimize the partitioning of the navigation domain into independent segments, under bandwidth and storage constraints. Experimental results confirm the potential of the proposed representation; namely, our system leads to similar compression performance as classical inter-view coding, while it provides the high level of flexibility that is required for interactive streaming. Hence, our new framework represents a promising solution for 3D data representation in novel interactive multimedia services

GPU-based photometric reconstruction from screen light

International audienceThis paper address the problem of obtaining from one web camera and a computer display, a facial reconstruction of the user within online messaging applications In this paper we present a 3D shape recovery in real time based on the photometric information of a set of 4 images under varying illumination conditions. Our GPU implementation provides online realistic 3D face recontructions

Joint disparity-motion estimation for coding of multiview video sequences

Dans cet article, nous proposons une technique d'estimation conjointe des champs de déplacements, présents dans une séquence d'images multi-vues, qui permet d'exploiter les différentes relations existantes entre ces champs. La stratégie adoptée permet de réduire le nombre de variables à estimer et augmenter ainsi la fiabilité et la précision des estimations. Le problème d'estimation conjointe est formulé comme un problème de programmation convexe, consistant à minimiser une fonction objectif convexe sur l'intersection d'ensembles convexes construits à partir des connaissances a priori et des observations. Les résultats présentés soulignent l'intérêt de l'estimation conjointe aussi bien en termes de reconstruction qu'en termes de cohérence des champs estimés

Arbitrarily shaped sub-block motion prediction in texture map compression using depth information

When transmitting the so-called “texture-plus-depth ” video format, texture and depth maps from the same viewpoint exhibit high cor-relation. Coded bits from one map can then be used as side infor-mation to encode the other. In this paper, we propose to use the depth information to divide the corresponding block in texture map into arbitrarily shaped regions (sub-blocks) for separate motion esti-mation (ME) and motion compensation (MC). We implemented our proposed sub-block motion prediction (MP) method for texture map coding using depth information as a new coding mode (z-mode) in H.264. Nonetheless, in practical experiments one can observe either a misalignment between texture and depth edges, or an aliasing ef-fect at the texture boundaries. To overcome this issue, z-mode offers two MC types: i) non-overlapping MC, and ii) overlapping MC. In the latter case, overlapped sub-blocks after ME are alpha-blended us-ing a properly designed filter. Moreover, the MV of each sub-block in z-mode is predicted using a Laplacian-weighted average of MVs of neighboring blocks of similar depth. Experimental results show that using z-mode, coding performance of the texture map can be improved by up to 0.7dB compared to native H.264 implementation at high bitrate. Index Terms — Depth-image-based rendering, motion predic-tion, arbitrary shape, video compression 1